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Abstract 

Net-centric  transformation  and  its  associated  practice  of  portfolio  management  require 
Department  of  Defense  (DOD)  managers  to  understand  the  effects  various  net-centric  command 
and  control  (C2)  services  have  on  operational  outcomes.  This  paper  discusses  an  approach 
developed  by  the  Johns  Hopkins  University  Applied  Physics  Laboratory  (JHU/APL)  to 
qualitatively  and  quantitatively  evaluate  the  technical,  functional,  and  mission  effectiveness  of 
C2  processes  and  services  in  a  complex,  hybrid  architectural  environment.  The  complexity  of 
that  environment  arises  from  the  need  for  legacy  C2  systems  and  newly-developed  net-centric 
processes  and  services  to  interoperate  in  a  common  environment.  The  JHU/APL  approach  uses 
scenarios  to  bound  the  mission  space  to  be  evaluated  and  employs  simulation  techniques  to 
represent  and  execute  the  scenarios  with  differing  levels  of  fidelity.  Simulation  types  include 
constructive,  virtual,  and  live  simulations.  The  technical  evaluation  results  obtained  from  the 
simulations  can  be  combined  with  estimated  deployment,  operations,  and  maintenance  costs  to 
facilitate  a  combined  technical/cost  comparison  among  service  offerings  from  competing 
portfolios.  Thus,  this  scenario-  and  simulation-based  evaluation  approach  is  expected  to  help 
DOD  managers  make  better-informed,  best-of-breed  decisions  regarding  which  net-centric  C2 
services  should  be  deployed  in  an  operational  environment. 
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1.  INTRODUCTION 


1.1  Background 

The  Department  of  Defense  (DOD)  has  embarked  on  a  path  to  make  force  transformation  an 
integral  element  of  national  defense  strategy.  Transformation  is  a  continuing  process  involving 
the  evolution  of  concepts,  processes,  organizations,  and  technologies.  The  term  “network-centric 
warfare”  is  applied  to  the  combination  of  emerging  and  evolving  tactics,  techniques,  and 
procedures  that  a  networked  force  can  employ  to  create  a  warfighting  advantage.  Network¬ 
centric  warfare  is  at  the  heart  of  force  transformation.  Successful  transformation  hinges  on 
making  the  right  investments  in  the  right  area  to  take  full  advantage  of  net-centric  warfare  and 
operations  technologies  and  practices.  (Garstka  and  Alberts  2004) 

The  tenets  of  net-centric  warfare  were  stated  in  the  DOD’s  2001  Network  Centric  Warfare 
Report  to  Congress : 

•  A  robustly  networked  force  improves  information  sharing 

•  Information  sharing  and  collaboration  enhances  the  quality  of  information  and  shared 
situational  awareness 

•  Shared  situational  awareness  enables  collaboration  and  self  synchronization,  and 
enhances  sustainability  and  speed  of  command 

•  These  in  turn  dramatically  increase  mission  effectiveness 
(DOD  2001) 


Net-centric  command  and  control  (C2)  services  are 
intended  to  help  achieve  information  and  decision 
superiority.  Today  operations  occur  in  a  complex 
environment  characterized  by  a  hybridization  of  net- 
centric  and  more  traditional  legacy  command  and 
control  capabilities  and  processes. 

Net-centric  transformation  and  its  associated  practice 
of  portfolio  management  require  DOD  decision¬ 
makers  to  understand  the  effects  various  net-centric 
command  and  control  services  have  on  operational 
outcomes.  Modeling  and  simulation  techniques  as  described  in  this  paper  can  be  adapted  to 
provide  a  foundation  for  assessing  net-centric  command  and  control  services. 

1.2  Purpose 

This  document  describes  an  approach  developed  by  the  Johns  Hopkins  University  Applied 
Physics  Laboratory  (JHU/APL)  to  qualitatively  and  quantitatively  evaluate  the  technical, 
functional,  and  mission  effectiveness  of  command  and  control  processes  and  services  in  a 
complex,  hybrid  architectural  environment  where  net-centric  and  legacy  capabilities  and 


The  evaluation  of  C2  issues  depends 
in  important  ways  on  both 
distinguishing  and  linking 
dimensional  parameters,  measures  of 
performance,  measures  of  C2 
effectiveness,  and  measures  of  force 
and  policy  effectiveness.  Modelling 
and  other  tools  must  be  designed  to 
support  this  requirement.  (NATO 
2002, 13-14) 
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processes  co-exist  and  must  interoperate.  This  approach  uses  scenarios  to  bound  the  mission 
space  to  be  evaluated  and  employs  simulation  techniques  using  multiple  levels  of  fidelity  or 
resolution  to  evaluate  net-centric  C2  in  that  complex  hybrid  environment. 

1.3  Scope 

This  paper  focuses  on  evaluating  strategic  and  national  echelons  of  command  and  control.  The 
scope  of  this  paper  includes  a  brief  discussion  of  the  challenges  of  net-centric  command  and 
control  and  the  goals  for  evaluating  net-centric  C2.  We  describe  a  framework  for  conducting  that 
evaluation  using  multi-resolution  modeling  (MRM).  The  paper  concludes  with  a  description  of  a 
limited  C2  evaluation  prototype  effort  (planned  for  FY05)  that  will  demonstrate  how  to  evaluate 
smart  agent  search  support  for  dynamically-created  communities  of  interest. 


2.  CHALLENGES  OF  NET-CENTRIC  C2 

Military  operations  take  place  in  environments  which  have  legacy  elements  that  are  platform¬ 
centric  and  transformed  elements  that  are  net-centric.  This  situation  is  likely  to  continue  for 
several  years.  The  contrasts  and  challenges  of  this  hybrid  environment  are  highlighted  in  Figure 
1. 
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Figure  1.  Command  and  Control  Operations  Occur  in  a  Hybrid  Environment 


Analyzing  command  and  control  performance  and  effectiveness  must  be  accomplished  in  the 
context  of  the  entire  chain  of  events  in  which  the  C2  activities  occur.  Modeling  and  simulation 
provide  techniques  to  facilitate  the  evaluation  of  new  C2  services  in  the  context  of  a  realistic 
operational  scenario.  The  scenario  defines  parameters  to  bound  the  evaluation. 

Our  hypothesis  is  that  net-centric  principles  advance  C2  capabilities.  Testing  that  hypothesis 
involves  addressing  these  questions: 


4 


•  What  are  the  performance  bounds  and  conditions? 

•  Where  is  net-centricity  appropriate  for  C2?  Where  is  it  inappropriate? 

•  How  does  net-centricity  affect  the  strategic  and  national  levels? 

The  expected  outcome  is  that  net-centric  C2  is  very  beneficial  in  most  cases  but,  perhaps,  not  in 
all. 

The  challenge  of  evaluating  net-centric  C2  is: 

•  To  develop  an  approach  to  qualitatively  and  quantitatively  evaluate  the  effectiveness  of 
net-centric  C2  processes  and  services  in  a  complex,  hybrid  architectural  environment,  and 

•  To  combine  those  results  with  lifecycle  costs  to  facilitate  better- informed  architecture  and 
technology  deployment  decisions  by  C2  portfolio  managers. 


3.  NET-CENTRIC  C2  EVALUATION  GOALS 

Command  and  control  as  defined  by  DOD  is 

The  exercise  of  authority  and  direction  by  a  properly  designated  commander  over 
assigned  and  attached  forces  in  the  accomplishment  of  the  mission.  Command  and 
control  functions  are  performed  through  an  arrangement  of  personnel,  equipment, 
communications,  facilities,  and  procedures  employed  by  a  commander  in  planning, 
directing,  coordinating,  and  controlling  forces  and  operations  in  the  accomplishment  of 
the  mission.  (DOD  2001  as  amended  through  30  November  2004,  101) 

While  this  definition  is  still  applicable  today,  the  way  in  which  C2  is  currently  being 
implemented  in  a  net-centric  Global  Information  Grid  (GIG)-enabled  environment  is  quite 
different  from  the  way  it  was  implemented  when  this  definition  was  created,  and  those 
differences  significantly  contribute  to  the  complexity  of  evaluating  net-centric  C2.  For  example, 
prior  to  the  net-centric  revolution,  C2  was  largely  achieved  via  the  use  of  stand-alone,  stove- 
piped  legacy  systems  that  communicated  with  one  another  via  point-to-point  network 
connections.  The  evaluations  of  those  systems  were  often  limited  to  demonstrating  that  the 
systems  met  their  requirements  and  were  able  to  effectively  communicate  with  one  another  over 
tightly  restricted  point-to-point  connections.  Today,  or  in  the  near  future,  commanders 
employing  net-centric  C2  implementations  will  take  advantage  of  distributed  computing  and 
communications  environments  that  involve  applications  developed  as  services  that  utilize 
scalable,  service-oriented  architectures.  Moreover,  those  net-centric  C2  services  must  interface  to 
some  degree  with  existing  legacy  C2  systems  since  the  legacy  systems  can’t  be  replaced  by  their 
net-centric  equivalents  instantaneously. 

The  complicated  hybrid  architecture  environment  and  the  highly-distributed  GIG  significantly 
contribute  to  the  complexity  of  evaluating  and  measuring  the  technical,  functional,  and  mission 
effectiveness  of  net-centric  C2  processes  and  services.  The  goals  of  the  C2  evaluation  approach 
described  in  this  paper  must  address  these  additional  levels  of  complexity.  Those  goals  are: 
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•  To  demonstrate  that  a  simulation-based  methodology  is  an  effective  means  for  evaluating 
command  and  control  in  a  hybrid  platform-centric  and  distributed  net-centric 
environment; 

•  To  demonstrate  that  constructive,  virtual,  and  live  simulation  techniques  can  effectively 
mitigate  some  of  the  challenges  of  evaluation  in  that  hybrid  environment; 

•  In  the  context  of  a  specific  operational  scenario, 

°  To  identify  how  and  where  the  application  of  net-centric  principles  augments  the 
effectiveness  of  existing  C2  capabilities 

°  To  identify  potential  gaps  where  the  application  of  net-centric  principles  fails  to 
augment  or  actually  degrades  C2  capabilities. 


4.  MULTI-RESOLUTION  MODELING  (MRM) 

4.1  Overview 

Simulation  is  used  to  construct  a  microcosm  of  the  capabilities  to  be  studied.  The  MRM 
approach  uses  two  or  more  models  of  different  fidelity  or  resolution  to  analyze  those  capabilities. 
MRM  uses  a  low- fidelity  model  to  provide  analytic  agility,  efficiency,  and  understanding  to  the 
problem  at  hand.  Often,  data  may  not  be  available  to  support  high-level  simulations  when 
decisions  must  be  made  on  the  value  of  developing  new  capabilities.  Decisions  must  be  made  in 
the  face  of  significant  uncertainties:  Will  the  desired  capabilities  be  achieved?  If  only  a  fraction 
of  the  estimated  improvement  is  achieved,  will  the  improvement  to  the  C2  process  still  be  worth 
the  investment?  Are  there  other  capabilities  that  would  provide  a  greater  improvement  with  less 
risk? 

Simulation  allows  us  to  ask  “What  if’  questions  and  evaluate  quantitative  data.  A  purely 
constructive  simulation  (i.e.,  a  model  where  all  aspects  of  the  system  are  simulated)  models  the 
process  and  allows  process  changes  to  be  evaluated  quickly  and  easily.  One  weakness  of  a 
constructive  simulation  is  that  it  is  limited  by  the  degree  to  which  the  simulated  process  is  a 
simplified  version  of  the  real  C2  process. 

To  reduce  this  limitation,  higher- fidelity  simulations  such  as  a  virtual  simulation  are  created. 
Virtual  simulations  inject  “humans-in-the-loop”  and  can  use  actual  hardware  and  software  as 
well.  Using  actual  subsystems  via  a  test  bed  interfaced  to  the  simulation  environment  removes 
the  simplifications  associated  with  the  constructive  simulation  and  results  in  a  higher-fidelity 
model.  The  virtual  simulation  allows  for  more  precise  analysis  of  the  real  capabilities  of  the 
system.  Having  humans  in  the  loop  allows  for  qualitative  assessments  to  be  conducted. 

Of  course,  C2  systems  exist  to  support  the  interactions  of  large  numbers  of  warfighters.  Live 
simulations  (e.g.,  exercises  or  war  games)  provide  significant  human-to-human  interactions  via 
the  C2  process  under  conditions  that  emulate  combat.  Analysis  of  these  events  provides  insight 
into  how  the  C2  process  would  contribute  to  mission  success  under  real-world  conditions. 
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Each  of  these  analysis  components  has  advantages  in  terms  of  cost,  time,  resources,  and  fidelity. 
By  combining  them  in  a  framework,  referred  to  in  this  paper  as  the  Multi-resolution  Modeling 
Evaluation  Framework  (MRMEF),  we  can  achieve  effective  analysis  of  C2  processes. 


4.2  Simulation  Description 

Three  types  of  simulations  can  be  used  to  analyze  C2.  Each  has  its  own  strengths  and 
weaknesses.  MRM  uses  a  low-resolution,  process-based  model  to  simulate  the  key  activities  of 
the  scenario.  Higher-fidelity  simulations  are  then  used  to  benchmark  the  low-fidelity  simulation. 

4. 2. 1  Constructive  Simulations 

Constructive  simulations  provide  the  basis  for  analysis  of  the  C2  process.  A  simulation  tool  such 
as  Arena  can  be  used  to  create  a  model  of  the  process  to  be  studied.  MRM  requires  a  specific 
approach  to  building  the  model.  If  possible,  a  hierarchical  structure  is  used  so  that  the  model  can 
use  a  low-fidelity  description  of  the  C2  process  to  do  exploratory  analysis.  The  model  should 
also  be  able  to  interface  with  the  higher-fidelity  hardware  and  software  to  simulate  specific 
aspects  of  the  process  in  higher  fidelity. 

For  example,  the  time  needed  to  choose  a  course  of  action  (COA)  could  be  modeled  as  a  simple 
random  variable  chosen  from  an  appropriate  distribution.  Such  a  function  is  the  lowest-fidelity 
model  of  the  COA  function.  Of  course,  COA  times  could  be  described  as  the  sum  of  several  sub¬ 
processes:  time  to  receive  instructions,  time  to  assemble  team,  time  to  collect  data,  time  to 
formulate  initial  COAs,  time  to  coordinate  and  refine  COAs,  time  to  approve  COAs.  Each  of 
these  subprocesses  could  be  described  as  a  function  of  even  more  variables.  Multiple  levels  of 
resolution  allow  the  overall  model  to  address  issues  appropriate  to  the  variables  and  processes 
described.  A  high-resolution  model  may  not  allow  simple  questions  to  be  answered  such  as 
“what  happens  if  the  opposing  forces  are  better  than  we  expect?”  There  may  be  27  variables  that 
describe  the  opponent’s  capabilities,  but  not  one  clear  and  reasonable  way  to  adjust  the 
effectiveness  of  the  opposing  forces.  Likewise,  a  low-resolution  model  is  not  likely  to  be  able  to 
address  the  issue  of  the  effect  of  improved  processing  speed  in  the  fire  control  radar  of  surface- 
to-air  batteries  on  the  outcome  of  a  theater  conflict. 

To  gain  an  understanding  of  the  impact  of  key  factors  in  the  process,  an  experimental  design  is 
often  created  and  data  are  gathered  on  the  performance  of  the  system  as  the  factors  change. 
Analysis  of  the  data  facilitates  development  of  mathematical  models  to  characterize  the  response 
of  the  system  to  changes  in  the  key  factors.  The  response  functions  often  illuminate  the  key 
issues  and  identify  trade-offs  and  optimal  solutions. 

Valid  MRM  will  often  require  stochastic  variables  represented  by  probability 
distributions,  not  merely  gross  measures  such  as  mean  values.  Further,  valid 
aggregate  models  must  sometimes  reflect  correlations  among  variables  that  might 
naively  be  seen  as  probabilistically  independent.  (Davis  and  Bigelow  1998,  x) 
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“Model-based  experimentation  has  its  most  obvious  utility  in  support  of  discovery  and 
hypothesis  testing  experiments.”  (Alberts  et  al.  2002,  319)  Reasons  for  choosing  a  model-based 
experiment  include: 


•  Efficiently  (in  terms  of  cost  and  time)  exploring  a  set  of  scenarios,  operational  concepts, 
and  technologies. . . . 

•  Supplementing  other  experiments . . . 

•  Exploring  the  potential  utility  of  technologies  that  may  not  yet  exist. 

(Alberts  et  al.  2002,  319-321) 

4.2.2  Virtual  Simulations 

Virtual  simulations  allow  a  high-fidelity  component  of  the  system  to  interact  with  the  lower- 
fidelity  constructive  simulation.  This  component  could  be  hardware,  software,  or  people.  Such 
experiments  are  often  called  “hardware-in-the-loop”,  “software-in-the-loop,  or  “human-in-the- 
loop”. 

Constructive  simulations  are  often  done  to  examine  the  feasibility  of  building  a  system.  Once  a 
prototype  has  been  constructed,  the  actual  system  can  perform  its  tasks  while  the  rest  of  the 
environment  is  simulated.  Such  an  experiment  replaces  specified  performance  parameters  with 
the  actual  performance  of  the  system  under  study.  Interfaces  to  the  constructive  simulation  must 
be  developed  and  used  to  create  the  necessary  translations  from  the  constructive  simulation  to  the 
virtual  simulation. 

MRM  allows  the  low-level  model  to  explore  the  analysis  space  and  identify  interesting  regions. 
These  regions  can  then  be  explored  by  creating  a  detailed  scenario  in  this  region  and  performing 
analysis  with  higher- fidelity  models  to  either  validate  the  findings  of  the  lower-fidelity  model  or 
to  provide  feedback  to  enhance  the  low-fidelity  model  so  that  it  is  a  better  representation  of  the 
process  under  study.  In  C2,  human  behavior  and  interactions  are  notoriously  difficult  to  model 
and  analyze.  A  key  advantage  of  applying  MRM  to  C2  is  that  the  low-level  model  provides 
understanding  of  the  impact  of  the  variables  in  the  C2  process  over  a  wide  range  of  potential 
values  for  those  variables.  The  high-level  modeling  effort  can  help  to  identify  what  the  true 
values  of  those  variables  are  for  specific  systems  and  scenarios. 

4.2.3  Live  Simulations 

Laboratories  and  simulations  are,  however,  far  less  realistic  than  the  settings  needed 
before  a  transformational  innovation  should  be  adopted.  Exercises,  whether  command 
post  (CPX)  or  field  training  exercises  (FTX),  represent  the  most  realistic  settings  for 
experimentation  and  should  be  employed  in  the  later  stages  of  experimentation 
campaigns.  (Alberts  et  al.  2002,  55-56) 

Accordingly,  one  part  of  our  multi-resolution  evaluation  framework  employs  live  simulations  to 
emulate  the  real-world  environment.  The  real  world  contains  many  human-to-human  interactions 
that  may  not  be  captured  in  the  lower-fidelity  simulations.  However,  live  simulations  are 
expensive,  time-consuming,  and  are  difficult  (if  not  impossible)  to  replicate.  A  live  simulation 
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provides  excellent  feedback  to  the  constructive  model  on  assumptions,  process  flows,  etc.  The 
outcome  of  the  live  simulation  should  be  predicted  by  the  lower-fidelity  models.  Deviations  from 
the  predictions  are  captured  and  studied.  In  this  way,  each  successive  level  of  increasing  model 
fidelity  provides  the  basis  for  improving  the  preceding  models. 

4.2.4  Evaluation  Metrics 

Experiments  are,  by  nature,  empirical.  Deciding  what  to  measure  is  critical  to  any  experiment. 
The  Code  of  Best  Practices  for  Experimentation  defines  a  metric  as  “the  application  of  a 
measurement  of  two  or  more  cases”.  (Alberts  et  al.  2002,  155-156) 

In  most  decisionmaking  problems,  it  is  necessary  to  define  several  measures  that 
together  provide  the  necessary  insights.  A  major  reason  for  this  is  that  a  single 
measure  may  not  provide  sufficient  scope  and/or  detail  to  analyse  the  impact  of 
specific  C2  elements,  particularly  second  and  third  order  effects  or  unintended 
consequences.  Many  analyses  are  conducted  precisely  in  order  to  enable  trade-off 
between  important  equities  which  can  only  be  seen  if  a  set  of  MoM  [Measures  of 
Merit]  is  generated  for  analysis.  The  set  of  MoM  selected  must  be  comprehensive 
to  ensure  that  all  factors  are  considered.  (NATO  2002,  94) 

Measures  of  Force  Effectiveness  (MoFEs)  are  the  ultimate  measures  of  military  success 
or  failure.  It  is  the  MoFEs  that  allow  trade-offs  between  disparate  capabilities  such  as 
more  tanks  vs.  more  planes.  Is  fielding  the  latest  generation  fighter  more  advantageous 
than  improving  the  C2  system?  The  behaviors  of  the  entities  in  question  can  be  measured 
and  are  considered  Measures  of  Performance  (MoPs).  “How  fast?”,  “How  far?”,  “How 
accurate?”,  “How  often?”,  etc.,  are  the  types  of  questions  measured  by  MoPs.  Measures 
of  Effectiveness  (MoEs)  estimate  the  impact  of  the  system  under  study  (and  its  associated 
MoPs)  with  measures  of  mission  effectiveness  such  as  number  of  days  to  establish  air 
supremacy,  blue/red  kill  ratios,  etc. 

In  MRM,  one  important  task  for  creating  higher-resolution  models  is  to  characterize  the 
behaviors  to  be  modeled  as  stochastic  variables  in  the  low-resolution  model.  If  the  mean 
and  variance  of  such  stochastic  variables  are  not  close  in  value  to  the  true  values  of  the 
underlying  process,  then  the  results  of  the  low-level  model  may  be  inaccurate, 
particularly,  if  one  or  more  of  those  stochastic  variables  contributed  significantly  to  the 
overall  outcome  of  the  mission.  One  of  the  primary  functions  of  the  higher-resolution 
model  is  to  validate  the  lower-resolution  model. 

4.3  Assess  Cost  and  Options 

Once  MoFEs  have  been  measured,  lifecycle  costs  can  be  analyzed  to  illuminate  the  cost-benefit 
trade-offs.  Figure  2  shows  an  example  of  what  such  a  cost-benefit  analysis  might  look  like. 
Imagine  six  possible  portfolios  of  C2  systems.  The  total  lifecycle  cost  of  each  portfolio  has  been 
estimated  and  the  MoFE  used  for  this  analysis  is  the  probability  of  mission  failure,  which  is 
plotted  on  the  horizontal  axis. 
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Figure  2.  Example  of  a  Cost-Benefit  Trade-off  Analysis 

In  this  type  of  multi-criteria  decision-making,  a  single  optimal  answer  doesn’t  exist  unless  the 
value  in  dollars  is  known  for  reducing  the  chance  of  mission  failure  by  one  percent.  For  example, 
if  mission  failure  implied  an  extended  operation  at  the  cost  of  $1B,  then  the  “value”  of  reducing 
the  probability  of  mission  failure  by  1%  is  estimated  to  be  $10M.  Among  the  possible  portfolios 
shown  above,  P3  is  estimated  to  cost  $150M  to  implement  and  deploy  and  has  a  99%  chance  of 
mission  success,  or,  alternatively,  a  1%  chance  of  mission  failure.  If  that  1%  risk  is  deemed 
acceptable,  then  selecting  P3  would  be  the  best  choice.  Further  reducing  the  risk  to 
approximately  0%  by  selecting  P4  would  not  be  cost  effective  because  the  cost  of  implementing 
and  deploying  P4,  i.e.,  $250M,  would  exceed  the  estimated  cost  of  a  1%  reduction  in  mission 
failure  by  a  factor  of  10.  Some  portfolios  can  be  identified  as  poor  choices:  PI  is  inferior  to  P2 
because  it  has  twice  the  chance  of  mission  failure  at  approximately  the  same  cost.  Both  P5  and 
P6  are  inferior  to  P3  because  they  cost  more  and  have  equivalent  or  higher  risk.  Portfolios  P2, 
P3,  and  P4  each  represent  reasonable  choices,  i.e.,  they  lie  along  the  “efficient  frontier”.  A 
decision-maker’s  preference  among  these  three  depends  upon  how  much  risk  is  considered 
acceptable  and  how  much  funding  is  available  to  achieve  that  level  of  risk. 

5.  MRM  EVALUATION  FRAMEWORK  (MRMEF)  APPROACH 
5.1  Description 

A  significant  challenge  to  evaluating  net-centric  C2  is  to  develop  an  approach  that  facilitates 
evaluation  of  C2  capabilities  in  a  complex  hybrid  architecture  environment.  Our  approach, 
referred  to  as  the  Multi-resolution  Modeling  Evaluation  Framework  (MRMEF),  uses 
constructive,  virtual,  and  live  simulations  and  hardware-,  software-,  and  humans-in-the-loop 
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where  appropriate.  Multi-resolution  Modeling  (MRM)  has  many  advantages  that  are  needed  to 
analyze  C2.  MRM  has  been  successful  because  it  has  the  characteristics  needed  to  solve  difficult 
analysis  problems  by  integrating  information  achieved  with  high-fidelity  models  and 
generalizing  the  results  and  implications  via  a  low-resolution  model  (Smith  1998).  An  overview 
of  the  MRMEF  is  shown  in  Figure  3. 


C2  Gaps  and 
Requirements 


Data-model, 
Ontology,  ... 


Scenario 
Mission  Thread 


C2  Processes 


Measures 


Virtual  Simulation  (Simulation  with  test  bed  involving  people  and  HW/SW  in-the-loop) 
Live  Simulation  (Simulation  with  real  components  in  an  exercise  environment) 


Figure  3.  Multi-resolution  Modeling  Evaluation  Framework 

The  simulation/exercise  environment  of  the  MRMEF  contains  the  entire  hardware  and  software 
infrastructure  needed  to  support  the  constructive,  virtual,  and  live  simulations  of  the  framework. 

The  “cube”  portion  of  the  diagram  represents  real  or  modeled  C2  or  C2 -related  components. 
Inputs  to  the  framework  consist  of  a  set  of  C2  services  to  be  evaluated;  the  services  were  derived 
from  C2  gap  analysis,  C2  requirements  definition,  data  modeling,  and  so  forth.  A  scenario 
defines  the  operational  mission,  i.e.,  the  problem  to  be  solved,  and  serves  as  the  contextual  basis 
for  the  evaluation.  Measures  to  assess  performance  and  effectiveness  are  defined  based  on  the 
context  of  the  scenario.  Evaluation  of  C2  capabilities  is  accomplished  by  executing  the  “cube” 
components,  (real,  simulated,  or  a  combination  of  real  and  simulated)  in  the  context  of  the 
appropriate  MRMEF  simulation/exercise  environment.  C2  evaluation  results  are  generated  as  a 
result  of  executing  the  scenario. 

An  “as-is”  evaluation  is  accomplished  by  developing  a  scenario-based  model  of  the  “as-is” 
process  to  be  evaluated  and  executing  that  model  as  a  constructive  simulation  within  the 
framework.  A  second  model  is  developed  representing  the  net-centric  equivalent  of  that  process. 
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The  net-centric  process,  which  may  involve  a  hybrid  of  legacy  and  net-centric  components,  both 
real  and  simulated,  is  executed  within  the  framework  as  a  virtual  simulation.  When  real 
components  are  used,  they  are  interfaced  with  the  simulation  via  a  separate  test  bed,  which 
allows  the  real  components  to  interact  as  necessary  with  modeled  components.  The  resulting 
simulation  executes  at  a  higher  level  of  fidelity  or  resolution  overall.  The  framework  also 
encompasses  a  very  high-fidelity  live  simulation  executed  outside  the  laboratory  environment 
with  real  players  and  components. 

Analysis  consists  of  comparing  the  “net-centric”  with  the  “as-is”  results  and  analyzing  the 
differences  to  determine,  both  qualitatively  and  quantitatively,  whether  the  application  of  net- 
centric  principles  and  components  to  an  existing  process  has  enhanced  or  degraded  engineering, 
command  and  control,  or  mission-level  performance  as  measured  via  MoPs,  MoEs,  and  MoFEs, 
respectively.  If  cost  information  about  deploying  and  maintaining  net-centric  C2  capabilities  is 
available  or  estimated,  those  data  can  be  combined  with  the  technical  evaluation  results  to  help 
guide  future  architecture,  acquisition,  and  deployment  decisions. 

The  steps  to  evaluate  a  proposed  net-centric  enhancement  to  C2  processes  are  summarized 
below: 

Step  0:  Identify  Components.  Inventory  the  elements  to  be  evaluated,  models,  datasets, 
and  other  data  and  tools  to  be  used. 

Step  1:  Define  Scenarios  and  Metrics.  Develop  or  leverage  upon  existing  scenarios  to 
bound  the  evaluation  problem  and,  within  that  context,  develop  a  set  of  metrics  to 
measure  the  performance  of  the  net-centric  C2  services  to  be  evaluated. 

Step  2:  Evaluate  via  Laboratory  Simulation.  Bounded  by  the  scenario,  evaluate  C2 
processes  and  net-centric  services  via  modeling  and  simulation  (constructive  and 
virtual).  Evaluate  both  the  proposed  net-centric  and  as-is  versions  of  the  processes. 
Different  models  may  be  required. 

Step  3:  Evaluate  via  Live  Simulation.  Deploy  and  evaluate  net-centric  services  from  one 
or  more  portfolios  via  live  simulation  in  an  exercise/experimentation  environment. 
Live  simulation  is  likely  to  be  used  to  focus  on  the  proposed  net-centric 
enhancements. 

Step  4:  Assess  Costs  and  Options.  Combine  technical  evaluation  results  with  expected 
deployment  and  operations  and  maintenance  costs  for  each  portfolio;  compare 
portfolio  offerings. 

5.2  MRMEF  Prototype 

We  plan  to  validate  the  MRMEF  concepts  by  constructing  an  evaluation  prototype  for  selected 
C2  capabilities.  Specifically,  we  will  demonstrate  how  to  evaluate  smart  agent  search  capabilities 
in  the  context  of  providing  information  provisioning  support  to  dynamic  operational 
communities  of  interest  (DOCOI)  collaboration  sessions.  By  demonstrating  that  evaluation 
capability,  we  will  propose  how  the  MRMEF  could  be  directly  extensible  to  the  evaluation  of 
other  existing  and  future  net-centric  C2  capabilities. 
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5.2.1  Prototype  Objectives 

The  objectives  of  this  prototype  effort  are  to: 

•  Demonstrate  how  the  MRMEF  is  used  to  evaluate  smart  agent  search  capabilities  for 
DOCOI  collaboration 

•  Show,  through  example,  how  the  MRMEF  could  be  directly  extensible  to  the  evaluation 
of  other  future  and  existing  net-centric  C2  capabilities 

5.3  Prototype  Approach 

The  approach  for  the  MRMEF  prototype  involves: 

•  Developing  or  utilizing  an  existing  scenario  in  which  DOCOIs  are  responsible  for 
developing  courses  of  action  to  accomplish  the  mission.  Figure  4  shows  a  sample  high- 
level  scenario.  The  scenario  includes  developing  COAs  for  integrated  missile  defense  in 
response  to  terrorists  acquiring  a  weapon  of  mass  destruction  (WMD)  and  potentially 
attempting  to  launch  it. 


Figure  4.  Sample  Integrated  Missile  Defense  Scenario 

•  Developing  a  set  of  metrics  to  evaluate  the  performance  of  the  DOCOIs  and  the 
supporting  smart  agent  search  capabilities.  Metrics  might  include  assessment  of  the  kinds 
of  participants  included  in  the  DOCOI,  their  role-based  access  levels,  their  level  of 
understanding  of  the  information  provided,  and  the  extent  to  which  understanding  was 
aligned  across  the  DOCOI. 

•  Developing  a  low-fidelity  simulation  to  characterize  the  current  (as-is)  C2  collaboration 
processes  in  the  context  of  the  scenario.  Measure  performance  via  the  metrics  defined 
above  using  a  simulation  tool  such  as  Arena. 

•  Developing  a  second  simulation  to  characterize  C2  collaboration  processes  in  a  net- 
centric  DOCOI  environment  augmented  by  smart  agent  search  capabilities.  Measure 
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performance  via  the  metrics  defined  above  in  a  simulation  environment  that  includes  the 
real  smart  agent  service  integrated  with  legacy  system  software  components,  where 
appropriate. 

•  Comparing  the  results  of  the  two  simulations  to  obtain  a  qualitative  and  quantitative 
assessment  of  the  smart  agent  search  capabilities  as  they  support  net-centric  C2  DOCOI 
collaboration  performance. 

5.3.1  Prototype  Assumptions  and  Constraints 

Although  the  MRMEF  is  intended  to  support  cost  analysis  as  well  as  performance  and 
effectiveness  evaluation,  that  function  will  not  be  included  in  the  FY05  prototype  effort.  The  cost 
analysis  capability  is  expected  to  be  evaluated  in  FY06. 


6.  CONCLUSIONS 

As  net-centric  C2  services  are  being  developed  in  the  context  of  service-oriented  architectures 
and  deployed  in  the  field  to  support  real-world  operations,  there  is  an  absolute  need  to  ensure 
these  services  operate  properly  in  both  a  unitary  mode  and  in  end-to-end  orchestrations  with 
other  services  and  systems.  The  benefits  of  applying  a  multi-resolution  modeling  approach  as 
described  in  this  paper  to  the  complex  problem  of  evaluating  services  and  legacy  systems  that 
need  to  interoperate  with  each  other  in  order  to  facilitate  effective  C2  are: 

•  Reduced  deployment  risks; 

•  Better  informed  architectural  and  deployment  decisions  by  DOD  managers; 

•  Increased  return  on  investment  via  reduced  operations  and  maintenance  costs; 

•  Enhanced  best-of-breed  selection  among  competing  portfolio  capabilities; 

•  Improved  techniques,  tactics,  and  procedures  and  concepts  of  operations  via  in-the-loop 
resource  experimentation;  and 

•  Synergism  with  the  Joint  C4I  Program  Assessment  and  Joint  Interoperability  Test 
Command  Testing  and  Certification  Processes. 
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Multi-Resolution  Modeling 
for  C2  Evaluation 


►  Transformation  requires  informed  decision  makers 

»  Cost/benefit  tradeoffs  within  the  C2  space 
»  Cost/benefit  tradeoffs  between  DoD  programs 

►  Challenge: 

»  To  qualitatively  and  quantitatively  evaluate  the 
effectiveness  of  net-centric  C2 

»  To  combine  those  results  with  service  life  cycle  costs 


We  need  to  be  able  to  estimate  the  improvement  in  military  outcomes  due 

to  potential  improvements  in  C2. 


C2  Processes  Exist  in  a  Hybrid 
Architecture  Environment 


Existing  and  evolving  C2  processes  must  interoperate  in 
hybrid  environments,  which  are: 


►  Legacy  Platform-Centric: 

»  System  or  Systems-of- 
Systems  Oriented 
Architectures 

»  Stovepipe  Characteristics: 
Low  Interoperability 

»  Limited  Standards 

»  Un-tagged  Data 

»  More  Traditional  TPED 
Approach 

»  Multi-networked;  Point-to- 
point  Connections 


►  Net-Centric: 

»  Service  Oriented 
Architectures 

»  Core  Services  (NCES) 

»  Service  Interoperability 

»  Well-defined  Standards 

»  Meta-tagged  Data 

»  Defined  Ontology 

»  Information  Exchange 
Cultural  Shift  (Power  to  the 
Edge;  TPPU) 

»  Global  Information  Grid 
(GIG) 
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APL  C2  Evaluation  Approach 


Step  1 :  Define  Scenario  and  Metrics:  Develop  or  leverage 
existing  scenarios  to  bound  the  evaluation  problem. 
Develop  a  set  of  metrics  to  measure  the  performance 
of  the  net-centric  C2  services  to  be  evaluated. 

Step  2:  Laboratory  Simulation:  In  the  context  of  a  scenario, 
evaluate  C2  processes  and  net-centric  services  via 
constructive  and  virtual  modeling  &  simulation. 

Step  3:  Live  Simulation:  Deploy  and  evaluate  net-centric 

services  from  one  or  more  portfolios  via  live  simulation 
in  an  exercise/experimentation  environment. 

Step  4:  Assess  Cost  and  Options:  Combine  technical 
evaluation  results  with  expected  deployment  and 
operations  &  maintenance  costs  for  each  portfolio; 
compare  portfolio  offerings. 
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Step  1 :  Define  Scenario  and  Metrics 


►  Measures  of  Force  Effectiveness  (MoFEs)  are  the  ultimate  measures  of 
military  success  or  failure 

►  Measures  of  Effectiveness  (MoEs)  measure  direct  impact 

►  Measures  of  Performance  (MoPs)  measure  the  system  characteristics 


Step  2:  Laboratory  Simulation 


►  Constructive  Simulation 

»  Low  fidelity 
»  Exploratory  analysis 
»  Identify  key  factors 

►  Virtual  Simulation 

»  “In-the-loop”  H/W,  S/W,  or  people 

»  Higher  fidelity  sub-modules 

»  Compare  and  analyze  the  net-centric  MoPs,  MoEs, 
and  MoFEs  of  that  evaluation 

»  Provide  feedback  to  constructive  simulation 
parameters  and  assumptions 


Step  3:  Live  Simulation 


►  Identify  an  appropriate  exercise/experimentation 
environment 

►  Develop  a  plan  to  deploy  and  evaluate  the  net- 
centric  services  from  each  portfolio  in  that 
environment 

►  For  each  portfolio  of  services 

»  Execute  the  scenario  as  a  live  simulation,  which 
includes  people,  real  services,  &  the  GIG 

»  Compare  and  analyze  the  net-centric  MoPs,  MoEs, 
and  MoFEs  of  that  evaluation  against  the  baseline 
data 

»  Generate  recommendations 


8 


Step  4:  Assess  Cost  and  Options 


►  For  each  portfolio,  combine  technical  evaluation 
results  with  life  cycle  costs 

►  Perform  a  trade  study  to  determine  which 
portfolio  offers  the  best  service  performance 
benefit  based  on  the  technical-to-cost  tradeoffs 


►  Identify  the  best-of- 
breed  choices 

►  Identify  poor  choices 
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Way-ahead  for  FY05 


►  Demonstrate  Multi-resolution  Modeling 
Evaluation  Framework  concepts  via  a  prototype 
to  evaluate  selected  C2  capabilities 

»  Focus  on  smart  agent  search  and  provisioning 
support  for  dynamically  created  communities  of 
interest 

►  Prototype  approach  (Step  1 ) 

»  Develop  a  scenario  based  on  the  sale  of  a  weapon  of 
mass  effect  (WME)  to  a  terrorist  organization  - 
completed 

»  Develop  a  set  of  metrics  to  evaluate  the  performance 
of  the  smart  agent  search  and  provisioning 
capabilities  -  in  progress 


Way-ahead  for  FY05  (continued) 


►  Prototype  approach  (Step  2) 

»  Current  C2  -  in  progress 

•  Develop  a  low-fidelity  simulation  of  the  current  (as-is)  C2 
collaboration  processes  in  the  context  of  the  scenario. 

•  Measure  performance  via  the  metrics  defined  above  using  a 
simulation  tool  such  as  Arena 

»  Net-Centric  C2 

•  Modify  the  low-fidelity  simulation  to  reflect  the  net-centric 
environment  augmented  by  smart  agent  search  and 
provisioning  capabilities. 

•  Measure  performance  via  the  metrics  defined  above  in  a 
simulation  environment  that  includes  the  smart  agent 
capabilities. 

»  Compare  the  results  of  the  two  simulations 

•  Qualitative  assessment 

•  Quantitative  assessment 
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Multi-resolution  Modeling  Enables 
Net-Centric  C2  Evaluation 


►  Better  informed  architectural  and  deployment  decisions 
by  DoD  managers 

►  Increased  return  on  investment  via  reduced  operations 
and  maintenance  costs 

►  Enhanced  best-of-breed  selection  among  competing 
portfolio  capabilities 

►  Improved  techniques,  tactics,  and  procedures  and 
concepts  of  operations  via  in-the-loop  resource 
experimentation 

►  Reduced  deployment  risks 
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Questions? 
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